Hybrid memory module with improved inter-memory data transmission path

ABSTRACT

Disclosed herein are techniques for implementing hybrid memory modules with improved inter-memory data transmission paths. The claimed embodiments address the problem of implementing a hybrid memory module that exhibits improved transmission latencies and power consumption when transmitting data between DRAM devices and NVM devices (e.g., flash devices) during data backup and data restore operations. Some embodiments are directed to approaches for providing a direct data transmission path coupling a non-volatile memory controller and the DRAM devices to transmit data between the DRAM devices and the flash devices. In one or more embodiments, the DRAM devices can be port switched devices, with a first port coupled to the data buffers and a second port coupled to the direct data transmission path. Further, in one or more embodiments, such data buffers can be disabled when transmitting data between the DRAM devices and the flash devices.

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/884,496, filed Oct. 15, 2015, now U.S. Pat. No. 10,241,727 issuedMar. 26, 2019, which is hereby incorporated in its entirety herein byreference.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent file or records, but otherwise reserves all copyrightswhatsoever.

FIELD

This disclosure relates to the field of memory systems and moreparticularly to techniques for a hybrid memory module with improvedinter-memory data transmission path.

BACKGROUND

As the massive volumes of electronically stored and transmitted data(e.g., “big data”) continue to increase, so does the need for electronicdata storage that is reliable and cost effective, yet quickly accessible(e.g., low latency). Specifically, more computing applications arerequiring that increasingly larger data sets be stored in “hot”locations for high speed access. Certain non-volatile memory (NVM)storage technologies, such as magnetic hard disk drives (HDDs), canprovide a reliable, low cost storage solution, yet with relatively highaccess latencies. Such storage technologies might be used for largevolumes of data in “cold” locations that are not often accessed (e.g.,data warehouses, archives, etc.). Other volatile or “dynamic” memorystorage technologies, such as dynamic random access memory (DRAM),provide lower access latencies, and might be used in “hot” locationsnear a computing host (e.g., CPU) to offer fast access to certain datafor processing. Yet, such storage technologies can have a relativelyhigh cost and risk of data loss (e.g., on power loss). Solid state NVM,such as Flash memory, can offer an improved form factor and accesslatency as compared to an HDD, yet still not approach the access latencyof DRAM.

In some cases, DRAM and Flash can be combined in a hybrid memory moduleto deliver the fast data access of the DRAM and the non-volatile dataintegrity (e.g., data retention) enabled by the Flash memory. One suchimplementation is the non-volatile dual in-line memory module (NVDIMM),which stores data in DRAM for normal operation, and stores data in Flashfor backup and/or restore operations (e.g., responsive to a power loss,system crash, normal system shutdown, etc.). Specifically, for example,the JEDEC standards organization has defined the NVDIMM-N product forsuch backup and/or restore applications. Many NVDIMM implementations canfurther be registered DIMMs (RDIMMs), which can use hardware registersand other logic, such as included in a registering clock driver (RDC),to buffer the address and control signals to the DRAM devices in orderto expand the capacity of the memory channels. Other NVDIMMimplementations can be load-reduced DIMMs (LRDIMMs), which can includedata buffers to buffer the data signals in order to reduce the loadingon the data bus and expand the capacity of the memory channels.

Unfortunately, legacy NVDIMM architectures can have functional andperformance limitations. Specifically, some NVDIMMs can exhibit longtransmission latencies and can have high power consumption whentransmitting data between the DRAM devices and the NVM devices duringdata backup and data restore operations. For example, some legacy NVDIMMarchitectures transmit data from the DRAM devices during such backup andrestore operations through data buffers used to reduce the loading onthe data bus during normal operation. Such an approach can result inlong transmission latencies through the data buffers and require thatthe data buffers remain powered on when power might be scarce (e.g.,during data backup after a power outage).

Techniques are needed to address the problems of implementing a hybridmemory module that exhibits improved transmission latencies and powerconsumption when transmitting data between the DRAM devices and the NVMdevices during data backup and data restore operations.

None of the aforementioned legacy approaches achieve the capabilities ofthe herein-disclosed techniques, therefore, there is a need forimprovements.

SUMMARY

The present disclosure provides an improved method, system, and computerprogram product suited to address the aforementioned issues with legacyapproaches. Specifically, the present disclosure provides a detaileddescription of techniques used in implementing a hybrid memory modulewith an improved inter-memory data transmission path (e.g., for databackup and restore). The claimed embodiments address the problem ofimplementing a hybrid memory module that exhibits improved transmissionlatencies and power consumption when transmitting data between DRAMdevices and NVM devices (e.g., flash devices) during data backup anddata restore operations. Some embodiments of the present disclosure aredirected to approaches for providing a direct data transmission pathcoupling a non-volatile memory controller and the DRAM devices totransmit data between the DRAM devices and the flash memory devices. Thedirect data transmission path eliminates the need for an indirect datatransmission path that traverses through a set of data buffersassociated with the DRAM devices and then to the non-volatile memorycontroller. In one or more embodiments, the DRAM devices can be portswitched devices, with a first port coupled to the data buffers and asecond port coupled to the direct data transmission path. Further, inone or more embodiments, such data buffers can be disabled whentransmitting data between the DRAM devices and the flash memory devices.

Further details of aspects, objectives, and advantages of the disclosureare described below and in the detailed description, drawings, andclaims. Both the foregoing general description of the background and thefollowing detailed description are exemplary and explanatory, and arenot intended to be limiting as to the scope of the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The drawings described below are for illustration purposes only. Thedrawings are not intended to limit the scope of the present disclosure.

FIG. 1A depicts an environment showing a hybrid memory module.

FIG. 1B1 is a diagrammatic representation of a direct data pathtechnique for improving data transmission throughput for data backup andrestore in hybrid memory modules, according to an embodiment.

FIG. 1B2 is a diagrammatic representation of a command replicationtechnique for improving data backup and restore throughput in hybridmemory modules, according to an embodiment.

FIG. 1B3 is a diagrammatic representation of a proprietary accesstechnique for enhancing non-volatile memory controller resource accessin hybrid memory modules, according to an embodiment.

FIG. 1B4 is a diagrammatic representation of a mode register settingsnooping technique for enhancing the programmability of the DRAM moderegister settings in a non-volatile memory controller control mode,according to an embodiment.

FIG. 2A is a schematic of a hybrid memory module.

FIG. 2B is a diagram showing interactions among hybrid memory modulecomponents pertaining to backup and restore events.

FIG. 3A is a schematic of hybrid memory module components showing anindirect data transmission path used for data backup and restore.

FIG. 3B is a schematic of a hybrid memory module showing a direct datatransmission path used for improving transmission latencies and powerconsumption during data backup and restore, according to someembodiments.

FIG. 4A depicts a command replicator subsystem used for improving databackup and restore throughput in hybrid memory modules, according tosome embodiments.

FIG. 4B presents a diagram showing interactions in hybrid memory modulesthat use a command replicator for improving data backup and restorethroughput, according to some embodiments.

FIG. 5A is a state diagram representing a command replication statemachine for accessing both sides of a DRAM array in hybrid memorymodules to improve data backup and restore throughput, according to someembodiments.

FIG. 5B is a state diagram representing a command replication statemachine for accessing multiple sides and/or multiple ranks of a DRAMarray in hybrid memory modules to improve data backup and restorethroughput, according to some embodiments.

FIG. 6A is a connection diagram of an independent connectionconfiguration for connecting DRAM devices to a non-volatile memorycontroller for backup, restore, and/or other operations.

FIG. 6B is a connection diagram of a first dual connection configurationhaving parallel A-side and B-side connections as implemented in hybridmemory modules with improved data backup and restore throughput,according to some embodiments.

FIG. 6C is a connection diagram of a second dual connectionconfiguration having dual parallel DRAM connections as implemented inhybrid memory modules with improved data backup and restore throughput,according to some embodiments.

FIG. 6D is a connection diagram of a quad connection configurationhaving quad parallel DRAM connections as implemented in hybrid memorymodules with improved data backup and restore throughput, according tosome embodiments.

FIG. 7A is a diagram of a proprietary access subsystem as implemented insystems for enhancing non-volatile memory controller resource access,according to some embodiments.

FIG. 7B illustrates a proprietary access protocol as used in hybridmemory modules for enhancing non-volatile memory controller resourceaccess, according to some embodiments.

FIG. 8A depicts a mode register controller subsystem used in hybridmemory modules for enhancing the programmability of the DRAM moderegister settings in a non-volatile memory controller control mode,according to some embodiments.

FIG. 8B is a diagram showing interactions in hybrid memory modules thatimplement a mode register controller for enhancing the programmabilityof the DRAM mode register settings in a non-volatile memory controllercontrol mode, according to some embodiments.

DETAILED DESCRIPTION

Embodiments of the present disclosure address problems attendant toelectronic data storage subsystem architectures (e.g., memory modules)that are exhibited in situations such as during data backup and restoreoperations.

Overview

Disclosed herein and in the accompanying figures are exemplaryenvironments, methods, and systems for hybrid memory modules withimproved inter-memory data transmission paths. More particularly,addressed herein are figures and discussions that teach techniques forcontrolling programmable data buffers and multiport DRAM devices toprovide a low power, direct path configuration for transmitting databetween DRAM devices and NVM devices during data backup and data restoreoperations in a hybrid memory module.

Definitions

Some of the terms used in this description are defined below for easyreference. The presented terms and their respective definitions are notrigidly restricted to these definitions—a term may be further defined bythe term's use within this disclosure.

-   -   The term “exemplary” is used herein to mean serving as an        example, instance, or illustration. Any aspect or design        described herein as “exemplary” is not necessarily to be        construed as preferred or advantageous over other aspects or        designs. Rather, use of the word exemplary is intended to        present concepts in a concrete fashion.    -   As used in this application and the appended claims, the term        “or” is intended to mean an inclusive “or” rather than an        exclusive “or”. That is, unless specified otherwise, or is clear        from the context, “X employs A or B” is intended to mean any of        the natural inclusive permutations. That is, if X employs A, X        employs B, or X employs both A and B, then “X employs A or B” is        satisfied under any of the foregoing instances.    -   The articles “a” and “an” as used in this application and the        appended claims should generally be construed to mean “one or        more” unless specified otherwise or is clear from the context to        be directed to a singular form.    -   The term “logic” means any combination of software or hardware        that is used to implement all or part of the disclosure.    -   The term “non-transitory computer readable medium” refers to any        medium that participates in providing instructions to a logic        processor.    -   A “module” includes any mix of any portions of computer memory        and any extent of circuitry including circuitry embodied as a        processor.

Reference is now made in detail to certain embodiments. The disclosedembodiments are not intended to be limiting of the claims.

Descriptions Of Exemplary Embodiments

FIG. 1A depicts an environment 1A00 showing a hybrid memory module. Asan option, one or more instances of environment 1A00 or any aspectthereof may be implemented in the context of the architecture andfunctionality of the embodiments described herein. Also, the environment1A00 or any aspect thereof may be implemented in any desiredenvironment.

As shown in FIG. 1A, environment 1A00 comprises a host 102 coupled to ahybrid memory module 120 through a system bus 110. The host 102 furthercomprises a CPU core 103, a cache memory 104, and a host memorycontroller 105. Host 102 can comprise multiple instances each of CPUcore 103, cache memory 104, and host memory controller 105. The host 102of environment 1A00 can further be based on various architectures (e.g.,Intel x86, ARM, MIPS, IBM Power, etc.). Cache memory 104 can bededicated to the CPU core 103 or shared with other cores. The hostmemory controller 105 of the host 102 communicates with the hybridmemory module 120 through the system bus 110 using a physical interface112 (e.g., compliant with the JEDEC DDR4 SDRAM standard, etc.).Specifically, the host memory controller 105 can write data to and/orread data from a first set of DRAM devices 124 ₁ and a second set ofDRAM devices 124 ₂ using a data bus 114 ₁ and a data bus 114 ₂,respectively. For example, the data bus 114 ₁ and the data bus 114 ₂ cantransmit the data as electronic signals such as a data signal, a chipselect signal, or a data strobe signal. The DRAM devices 124 ₁ and/orthe DRAM devices 124 ₂ might each comprise an array of eight or nineDDR4 memory devices (e.g., SDRAM) arranged in various topologies (e.g.,A/B sides, single-rank, dual-rank, quad-rank, etc.). Other memorydevices (e.g., DDR3 memory devices) can comprise the DRAM devices. Insome cases, as shown, the data to and/or from the DRAM devices 124 ₁ andthe DRAM devices 124 ₂ can be buffered by a set of data buffers 122 ₁and data buffers 122 ₂, respectively. Such data buffers can serve toboost the drive of the signals (e.g., data or DQ signals, etc.) on thesystem bus 110 to help mitigate high electrical loads of large computingand/or memory systems.

Further, commands from the host memory controller 105 can be received bya command buffer 126 (e.g., registering clock driver or RCD) at thehybrid memory module 120 using a command and address (CA) bus 116. Forexample, the command buffer 126 might be a registering clock driver(RCD) such as included in registered DIMMs (e.g., RDIMMs, LRDIMMs,etc.). Command buffers such as command buffer 126 can comprise a logicalregister and a phase-lock loop (PLL) to receive and re-drive command andaddress input signals from the host memory controller 105 to the DRAMdevices on a DIMM (e.g., DRAM devices 124 ₁, DRAM devices 124 ₂, etc.),reducing clock, control, command, and address signal loading byisolating the DRAM devices from the host memory controller 105 and thesystem bus 110. In some cases, certain features of the command buffer126 can be programmed with configuration and/or control settings.

The hybrid memory module 120 shown in FIG. 1A further comprises anon-volatile memory controller 128 coupled to a flash controller 132 anda set of flash memory devices 134. The presence of the flash memorydevices 134 (e.g., NAND flash memory chips) and the DRAM devices on adual in-line memory module (DIMM), in part, defines the “hybrid”characteristic of the hybrid memory module 120, at least according toJEDEC. Such hybrid memory modules can be referred to as non-volatileDIMMs (NVDIMMs), and can appear as a DRAM DIMM to the system controller(e.g., host memory controller 105) and/or share a memory channel withother DRAM DIMMs. For example, JEDEC has identified three NVDIMMconfigurations as follows:

-   -   NVDIMM-N: A hybrid memory module consisting of DRAM made        persistent through the use of Flash memory. No Flash memory        beyond that needed for persistence operations (e.g., data        backup, data restore, etc.) is accessible by the host memory        controller.    -   NVDIMM-P: A hybrid memory module consisting of DRAM made        persistent through the use of Flash memory. Flash memory beyond        that needed for persistence is accessible by the host memory        controller as a block-oriented mass storage device.    -   NVDIMM-F: A hybrid memory module consisting of Flash memory        accessed by the host memory controller as a block-oriented mass        storage device.

The hybrid memory module 120 shown in environment 1A00 can be consideredan NVDIMM-N configuration. As such, a backup power module 150 is showncoupled to the hybrid memory module 120 to deliver power to the hybridmemory module 120 during persistence operations such as data backup anddata restore in the event of a system power loss. For example, thebackup power module 150 might comprise super capacitors (e.g.,supercaps) and/or battery packs attached to the hybrid memory module 120via a tether cable and store enough charge to keep at least a portion ofthe hybrid memory module 120 powered up long enough to copy all of itsdata from the DRAM to the flash memory.

Further, the hybrid memory module 120 shown in environment 1A00 presentsmerely one partitioning. The specific example shown where the commandbuffer 126, the non-volatile memory controller 128, and the flashcontroller 132 are separate components is purely exemplary, and otherpartitioning is reasonable. For example, any or all of the componentscomprising the hybrid memory module 120 and/or other components cancomprise one device (e.g., system-on-chip or SoC), multiple devices in asingle package or printed circuit board, multiple separate devices, andcan have other variations, modifications, and alternatives.

Unfortunately, legacy NVDIMM architectures can have functional andperformance limitations. Specifically, some NVDIMMs can exhibit longlatencies and low throughput during certain operations, such as thosepertaining to data backup and/or data restore operations. The hereindisclosed techniques address such limitations and other legacy issues asdescribed in the following and throughout.

FIG. 1B1 is a diagrammatic representation of a direct data pathtechnique 1B100 for improving data transmission throughput for databackup and restore in hybrid memory modules. As an option, one or moreinstances of direct data path technique 1B100 or any aspect thereof maybe implemented in the context of the architecture and functionality ofthe embodiments described herein. Also, the direct data path technique1B100 or any aspect thereof may be implemented in any desiredenvironment.

As shown in FIG. 1B1, the direct data path technique 1B100 is depictedin the environment 1A00 comprising the hybrid memory module 120. Thedirect data path technique 1B100 can address the problems attendant toimplementing a hybrid memory module that exhibits improved transmissionlatencies and power consumption when transmitting data between themodule DRAM devices and the module NVM devices during data backup anddata restore operations. Specifically, in some embodiments, the directdata path technique 1B100 comprises a direct data transmission path 162coupling the non-volatile memory controller 128 and the DRAM devices 124₁ and the DRAM devices 124 ₂. The non-volatile memory controller 128 canuse the direct data transmission path 162 to transmit data between theDRAM devices and the flash memory devices 134, eliminating the need fora path coupling the data buffers (e.g., data buffers 122 ₁, data buffers122 ₂) and the non-volatile memory controller 128. In some embodiments,the DRAM devices can be port switched devices, each comprising a firstport (e.g., first port 164 ₁, first port 164 ₂) coupled to the data bus(e.g., data bus 114 ₁, data bus 114 ₂), and a second port (e.g., secondport 166 ₁, second port 166 ₂) coupled to the direct data transmissionpath 162, such that the first port is disabled and the second port isenabled when transmitting data between the DRAM devices and the flashmemory devices. Further, in one or more embodiments, the data buffers(e.g., data buffers 122 ₁, data buffers 122 ₂) can be disabled whentransmitting data between the DRAM devices and the flash memory devices.

FIG. 1B2 is a diagrammatic representation of a command replicationtechnique 1B200 for improving data backup and restore throughput inhybrid memory modules. As an option, one or more instances of commandreplication technique 1B200 or any aspect thereof may be implemented inthe context of the architecture and functionality of the embodimentsdescribed herein. Also, the command replication technique 1B200 or anyaspect thereof may be implemented in any desired environment.

As shown in FIG. 1B2, the command replication technique 1B200 isdepicted in the environment 1A00 comprising the hybrid memory module120. The command replication technique 1B200 can address the problemsattendant to implementing a hybrid memory module that overcomes thethroughput limitations of the non-volatile memory controller (NVC)communications interface used for DRAM read and write commands duringdata backup and data restore operations. Specifically, in someembodiments, the command buffer 126 can receive host commands 171 fromthe host memory controller 105, receive local commands 172 from thenon-volatile memory controller 128, and issue DRAM commands (e.g., DRAMcommands 174 ₁, DRAM commands 174 ₂) to the DRAM devices (e.g., DRAMdevices 124 ₁, DRAM devices 124 ₂). Further, the command replicationtechnique 1B200 can comprise a command replicator 176 (e.g., implementedin the command buffer 126) to generate command sequences comprisingreplicated DRAM commands (e.g., replicated DRAM commands 178 ₁,replicated DRAM commands 178 ₂) to be issued by the command buffer 126to the DRAM devices. In one or more embodiments, the command sequence isbased at least in part on the local commands 172 received by the commandbuffer 126 (e.g., during an NVC control mode).

In some embodiments, the command sequence is issued to the DRAM devicesby the command buffer 126 responsive to receiving one or more instancesof the local commands 172. In one or more embodiments, the commandsequence can comprise wait times between the replicated DRAM commands.Further, in some embodiments, the replicated DRAM commands can accessone or more memory locations (e.g., sides, ranks, bytes, nibbles, etc.)of the DRAM devices. Also, in other embodiments, the command sequencecan comprise sets of replicated DRAM commands that access respectiveportions of the DRAM devices (e.g., two sets of commands to access twogroups of DRAM devices sharing a connection).

FIG. 1B3 is a diagrammatic representation of a proprietary accesstechnique 1B300 for enhancing non-volatile memory controller resourceaccess in hybrid memory modules. As an option, one or more instances ofproprietary access technique 1B300 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the proprietary access technique1B300 or any aspect thereof may be implemented in any desiredenvironment.

As shown in FIG. 1B3, the proprietary access technique 1B300 is depictedin the environment 1A00 comprising the hybrid memory module 120. Theproprietary access technique 1B300 can address the problems attendant toimplementing a hybrid memory module that expands the non-volatile memorycontroller (NVC) resource access, yet does not impact host memorycontroller resource access, when in a host control mode. Specifically,in some embodiments, the command buffer 126 can receive host commands171 from the host memory controller 105, and receive local commands 172from the non-volatile memory controller 128. In some cases, suchcommands are interpreted by a set of control setting access logic 181 toaccess a set of control setting registers 182 that hold certaininstances of control settings (e.g., used to adjust certaincharacteristics of the command buffer 126). Further, the control settingregisters 182 can comprise a protected register space 185 not accessibleby the non-volatile memory controller 128 in the host control mode.

In one or more embodiments, the proprietary access technique 1B300comprises a proprietary access engine 184 to interpret one or moreproprietary access commands 188 from the non-volatile memory controller128 to access the protected register space 185 while still in the hostcontrol mode. In one or more embodiments, the proprietary access engine184 comprises a set of proprietary control setting access logic based inpart on the control setting access logic 181 to interpret theproprietary access commands 188 to write to and/or read from theprotected register space 185. In one or more embodiments, theproprietary access engine 184 comprises a command router to route thelocal commands 172 to the control setting access logic 181 and route theproprietary access commands 188 to the proprietary control settingaccess logic. In one or more embodiments, the proprietary accesscommands 188 are routed to the proprietary control setting access logicbased at least in part on a proprietary mode triggered by a sequence oflocal commands. Further, in some embodiments, the proprietary accessengine 184 comprises an access arbiter to allow access to the protectedregister space 185 invoked by the host commands 171 and/or theproprietary access commands 188.

FIG. 1B4 is a diagrammatic representation of a mode register settingsnooping technique 1B400 for enhancing the programmability of the DRAMmode register settings in a non-volatile memory controller control mode.As an option, one or more instances of mode register setting snoopingtechnique 1B400 or any aspect thereof may be implemented in the contextof the architecture and functionality of the embodiments describedherein. Also, the mode register setting snooping technique 1B400 or anyaspect thereof may be implemented in any desired environment.

As shown in FIG. 1B4, the mode register setting snooping technique 1B400is depicted in the environment 1A00 comprising the hybrid memory module120. The mode register setting snooping technique 1B400 can address theproblems attendant to implementing a hybrid memory module that exhibitsenhanced programmability of the DRAM mode register settings in anon-volatile memory controller control mode (e.g. NVC control mode),such as when invoked for data backup and data restore operations.Specifically, in some embodiments, the command buffer 126 can receivehost commands 171 from the host memory controller 105, receive localcommands 172 from the non-volatile memory controller 128, and issue DRAMcommands to the DRAM devices (e.g., DRAM devices 124 ₁, DRAM devices 124₂). For example, such DRAM commands can be mode register setting (MRS)commands issued by the host memory controller 105 (e.g., during hostcontrol mode) and/or by the non-volatile memory controller 128 (e.g.,during NVC control mode). Further, the mode register setting snoopingtechnique 1B400 can comprise a mode register controller 192 to capture(e.g., “snoop”) a set of captured mode register settings 194 from thehost commands 171 and generate certain generated mode register settingcommands (e.g., generated mode register setting commands 196 ₁,generated mode register setting commands 196 ₂) based on the capturedmode register settings 194. In one or more embodiments, the generatedmode register setting commands can be issued to the DRAM devices by thecommand buffer 126 responsive to receiving certain instances of thelocal commands 172. In some embodiments, the captured mode registersettings 194 can be modified to produce one or more modified capturedmode register settings to be used to generate the generated moderegister setting commands.

Further details pertaining the aforementioned techniques forhigh-throughput low-latency hybrid memory modules are disclosed in thefollowing and herein.

FIG. 2A is a schematic of a hybrid memory module 2A00. As an option, oneor more instances of hybrid memory module 2A00 or any aspect thereof maybe implemented in the context of the architecture and functionality ofthe embodiments described herein. Also, the hybrid memory module 2A00 orany aspect thereof may be implemented in any desired environment.

The hybrid memory module 2A00 is one example of an NVDIMM configuration.Specifically, the DRAM devices of the hybrid memory module 2A00 comprise18 DDR4 devices (e.g., ten instances of DRAM devices 124 ₁ and eightinstances of DRAM devices 124 ₂) having data signals (e.g., DQ, DQS,etc.) delivered to a DDR4 DIMM edge connector 202 through a plurality ofdata buffers (e.g., five instances of data buffers 122 ₁ and fourinstances of data buffers 122 ₂). In some cases, two DDR4 devices canshare the high bit rate MDQ/MDQS signal connections to a respective databuffer (e.g., DB02 device) in a parallel configuration. Further, a firstportion of the DDR4 devices (e.g., DDR4-0 to DDR4-4, and DDR4-9 toDDR4-13) can comprise an A-side of the DRAM configuration, and a secondportion of the DDR4 devices (e.g., DDR4-5 to DDR4-8 and DDR4-14 toDDR4-17) can comprise a B-side of the DRAM configuration. In some cases,such configurations can be detected by a serial presence detector or SPDat module initialization. The non-volatile memory controller 128 canfurther have access to the DDR4 device data signals through an LDQ/LDQSpath between the data buffers and the “DRAM Interface” of thenon-volatile memory controller 128.

As shown, the command buffer 126 can receive commands, addresses, andother information through the DDR4 DIMM edge connector 202 at an inputcommand/address or C/A interface. The command buffer 126 can furthercommunicate (e.g., receive local commands) with the non-volatile memorycontroller 128 using a local communications interface supporting aphysical layer communications protocol such as the LCOM interfaceprotocol defined by JEDEC. The command buffer 126 can communicate (e.g.,forward DRAM commands) with the DDR4 devices using an outputcontrol/address/command interface (e.g., see the QA output signals forcommunicating with the A-side, and the QB output signals forcommunicating with the B-side). In some cases, the command buffer 126can also communicate (e.g., send control setting commands) with the databuffers using a data buffer control/communication or BCOM interface.Other signals shown in FIG. 2A include those pertaining to the I2Cserial bus and the Save_n memory system signal (e.g., for invoking abackup operation at power loss).

The foregoing signals, interfaces, connections, and other components ofthe hybrid memory module 2A00 can be used to execute backup and restoreoperations as discussed in FIG. 2B.

FIG. 2B is a diagram showing interactions among hybrid memory modulecomponents 2B00 pertaining to backup and restore events. As an option,one or more instances of interactions among hybrid memory modulecomponents 2B00 or any aspect thereof may be implemented in the contextof the architecture and functionality of the embodiments describedherein. Also, the interactions among hybrid memory module components2B00 or any aspect thereof may be implemented in any desiredenvironment.

As shown in FIG. 2B, the interactions among hybrid memory modulecomponents 2B00 specifically pertain to interactions among the earlierdescribed components comprising the host memory controller 105, thecommand buffer 126, the non-volatile memory controller 128, thecollective set of DRAM devices 124, and the flash memory devices 134.Such components can exhibit a set of high-level interactions (e.g.,operations, messages, etc.) as shown. Specifically, the interactions canpertain to backup and restore operations executed on a hybrid memorymodule. As shown, the host memory controller 105 might have control(e.g., in a host control mode) so as to issue DRAM commands to thecommand buffer 126 (see message 252 ₁) that might be forwarded to theDRAM devices 124 (see message 253 ₁). In some cases, the DRAM commandscan result in read and/or write data transferred between the host memorycontroller 105 and the DRAM devices 124 (see message 2540.

Such activity might continue until a data backup event signal isreceived at the non-volatile memory controller 128 (see operation 256).For example, the host and/or the hybrid memory module might havedetected the loss of power and triggered the data backup event. Suchbackup events can be invoked at the non-volatile memory controller 128from the host memory controller 105 (e.g., via the command buffer 126),from the Save_n signal, and from the I2C bus. In response, control canbe provisioned to the non-volatile memory controller 128 by, forexample, writing to certain control register settings of the commandbuffer 126 (see message 2580. The backup operation might then commencewith the non-volatile memory controller 128 sending new mode registersettings (e.g., specific to the backup operation) to the command buffer126 (see message 260) that can be forwarded to the DRAM devices 124 (seemessage 261). The non-volatile memory controller 128 can then begin toissue backup commands to the command buffer 126 (see message 262) thatcan be forwarded to the DRAM devices 124 (see message 263) to save datafrom the DRAM devices 124 to the flash memory devices 134 (see message264). Such backup interactions can continue in a loop (see loop 266)until the backup operation is complete (e.g., all data is saved).

After a time lapse 268, a data restore event signal might be received bythe non-volatile memory controller 128 (see operation 270). For example,the line power to the computing system might have returned to triggerthe data restore event. In response, control can be provisioned to thenon-volatile memory controller 128 by, for example, writing to certaincontrol register settings of the command buffer 126 (see message 258 ₂).The restore operation might commence with the non-volatile memorycontroller 128 sending new mode register settings (e.g., specific to therestore operation) to the command buffer 126 (see message 274) that canbe forwarded to the DRAM devices 124 (see message 275). The non-volatilememory controller 128 can then begin to issue restore commands to thecommand buffer 126 (see message 276) that can be forwarded to the DRAMdevices 124 (see message 278) to restore data from the flash memorydevices 134 to the DRAM devices 124 (see message 280). Such restoreinteractions can continue in a loop (see loop 281) until the restoreoperation is complete (e.g., all data is restored).

When the restore is complete, the command buffer 126 can provisioncontrol to the host memory controller 105 (see message 282). The hostmemory controller 105 might then initialize the host control session bysending new mode register settings (e.g., specific to host operations)to the command buffer 126 (see message 284) that can be forwarded to theDRAM devices 124 (see message 285). The host memory controller 105 canthen resume memory access operations by issuing DRAM commands to thecommand buffer 126 (see message 252 ₂) to be forwarded to the DRAMdevices 124 (see message 253 ₂) to invoke, in some cases, the transferof read and/or write data between the host memory controller 105 and theDRAM devices 124 (see message 254 ₂).

The hybrid memory module 2A00 and the interactions among hybrid memorymodule components 2B00 exemplify various limitations addressed by theherein disclosed techniques. Specifically, FIG. 3B describes the hereindisclosed for improving (e.g., as compared to FIG. 3A) transmissionlatencies and power consumption when transmitting data between the DRAMdevices and the flash memory devices during data backup and data restoreoperations.

FIG. 3A is a schematic of hybrid memory module components 3A00 showingan indirect data transmission path used for data backup and restore. Asan option, one or more instances of hybrid memory module components 3A00or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the hybrid memory module components 3A00 or any aspect thereof maybe implemented in any desired environment.

FIG. 3A shows the hybrid memory module 2A00 comprising a highlightedinstance of an indirect data path 302 used for data backup and datarestore operations. As shown, the indirect data path 302 requires thatdata from the DRAM devices (e.g., DDR4 devices) be routed through thedata buffers (e.g., DB02 devices) to the DRAM interface of thenon-volatile memory controller 128 during backup and restore. Theindirect data path 302 comprises two segments: the MDQ/MDQS high bitrate host path between the DRAM devices and the data buffers, and thelow bit rate LDQ/LDQS path between the data buffers and the DRAMinterface of the non-volatile memory controller 128. Such an indirectdata path 302 requires that the DRAM devices, the command buffer 126,and the data buffers run at the high bit rate of the host path,consuming unnecessary power. Further, such components might also need torely on the host training settings to comply with signal timingrequirements. The foregoing issues pertaining to the indirect data path302 are addressed by the herein disclosed techniques as described inFIG. 3B.

FIG. 3B is a schematic of a hybrid memory module 3B00 showing a directdata transmission path used for improving transmission latencies andpower consumption during data backup and restore. As an option, one ormore instances of hybrid memory module 3B00 or any aspect thereof may beimplemented in the context of the architecture and functionality of theembodiments described herein. Also, the hybrid memory module 3B00 or anyaspect thereof may be implemented in any desired environment.

The hybrid memory module 3B00 shown in FIG. 3B comprises a direct datatransmission path 162 as described herein. The direct data transmissionpath 162 addresses the problems attendant to implementing a hybridmemory module that exhibits improved transmission latencies and powerconsumption when transmitting data between the DRAM devices and the NVMdevices during data backup and data restore operations. Specifically,the direct data transmission path 162 provides a low-speed direct pathbetween the DRAM devices and the non-volatile memory controller 128(e.g., at the LDQ/LDQS DRAM interface) for use during data backup anddata restore operations. In one or more embodiments, the high bit ratehost path can be coupled to a first port (e.g., data port A) of a portswitched DRAM device, and the direct data transmission path 162 can becoupled to a second port (e.g., data port B) of the port switched DRAMdevice. In some cases, a portion of the direct data transmission path162 can be shared by two or more DRAM devices. Such a low-speed directpath can have several advantages, such as enhanced signal integrity,improved timing margins, eliminated timing training, lower latency(e.g., no DB02 devices in path), and/or other advantages. The directdata transmission path 162 further allows, during backup and restore,the DRAM devices and the command buffer 126 to be in a DLL off mode, andthe data buffers (e.g., DB02 devices) to be in a clock stoppedpower-down mode. Such allowed modes can reduce power consumption, reducereconfiguration time when switching to/from backup and/or restore,and/or other benefits.

FIG. 4A depicts a command replicator subsystem 4A00 used for improvingdata backup and restore throughput in hybrid memory modules. As anoption, one or more instances of command replicator subsystem 4A00 orany aspect thereof may be implemented in the context of the architectureand functionality of the embodiments described herein. Also, the commandreplicator subsystem 4A00 or any aspect thereof may be implemented inany desired environment.

The command replicator subsystem 4A00 shown in FIG. 4A addresses theproblems attendant with implementing a hybrid memory module thatovercomes the throughput limitations of the non-volatile memorycontroller (NVC) communications interface used for DRAM read and writecommands during data backup and data restore operations. Specifically,the local commands 172 issued from the non-volatile memory controller128 to the command buffer 126 using the LCOM interface (e.g., duringdata backup and data restore operations) limit the throughput of thedata transfer to the LCOM interface command rate. For example, DRAMcommands (e.g., DRAM commands 174 ₁, DRAM commands 174 ₂) issued aslocal commands 172 through the LCOM interface can require 16 LCOM LCKclock cycles per command. As a comparison, DRAM commands issued as hostcommands 171 from the host memory controller 105 can require four orfive DRAM CK clock cycles per command. The command replicator subsystem4A00 can overcome this limitation by using the command replicator 176comprising a state machine 402 to generate sequences of replicated DRAMcommands (e.g., replicated DRAM commands 178 ₁, replicated DRAM commands178 ₂) to issue to the DRAM devices in response to respective instancesof local commands 172. In one or more embodiments, the state machine 402can operate at the DRAM clock rate (e.g., CK clock) to enable fastcommand replication. In one or more embodiments, the replicated DRAMcommands can further be generated based in part on various options, suchas related to addressing alternate sides, addressing multiple ranks,automatically incrementing addresses, and/or other options.

The command replicator subsystem 4A00 presents merely one partitioning.The specific example shown is purely exemplary, and other partitioningis reasonable. A technique for applying such systems, subsystems, andpartitionings to data backup and data restore operations according tothe herein disclosed techniques is shown in FIG. 4B.

FIG. 4B presents a diagram showing interactions in hybrid memory modules4B00 that use a command replicator for improving data backup and restorethroughput. As an option, one or more instances of interactions inhybrid memory modules 4B00 or any aspect thereof may be implemented inthe context of the architecture and functionality of the embodimentsdescribed herein. Also, the interactions in hybrid memory modules 4B00or any aspect thereof may be implemented in any desired environment.

As shown in FIG. 4B, the interactions in hybrid memory modules 4B00pertain to interactions among the earlier described componentscomprising the host memory controller 105, the command buffer 126, thenon-volatile memory controller 128, the collective set of DRAM devices124, and the flash memory devices 134, according to the herein disclosedtechniques for overcoming the throughput limitations of the non-volatilememory controller LCOM interface used for DRAM read and write commandsduring data backup and data restore operations. In one or moreembodiments, the interactions in hybrid memory modules 4B00 can beimplemented using the command replicator subsystem 4A00. As shown, thenon-volatile memory controller 128 might receive a data backup eventsignal (see operation 412) triggered, for example, by a detected powerloss. In such a case, control can be provisioned to the non-volatilememory controller 128 at the command buffer 126 (see message 414 ₁). Thenon-volatile memory controller 128 might then invoke the backupoperation by sending a backup command to the command buffer 126 (seemessage 416). For example, the backup command might be delivered usingthe LCOM interface. The command buffer 126 can replicate the backupcommand (see operation 418) according to the herein disclosed techniquesand issue the replicated commands to the DRAM devices 124 (see message420) to save data from the DRAM devices 124 to the flash memory devices134 (see message 422). For example, one LCOM read command can invoke theexecution of multiple replicated read commands (e.g., reading a full rowof memory), thus increasing the throughput as compared to issuing eachread command through the LCOM interface. Such backup interactions withreplication can continue in a loop (see loop 424) until the backupoperation is complete (e.g., all data is saved).

After a time lapse 468, the non-volatile memory controller 128 mightreceive a data restore event signal (see operation 426). For example,the line power to the computing system might have returned to triggerthe data restore event. In such cases, control can be provisioned to thenon-volatile memory controller 128 at the command buffer 126 (seemessage 414 ₂). The non-volatile memory controller 128 might then invokethe restore process by issuing a restore command to the command buffer126 (see message 428). For example, the restore command might bedelivered using the LCOM interface. The command buffer 126 can replicatethe restore command (see operation 430) according to the hereindisclosed techniques and issue the replicated commands to the DRAMdevices 124 (see message 432) to restore data from the flash memorydevices 134 to the DRAM devices 124 (see message 434). For example, oneLCOM write command can invoke the execution of multiple replicated writecommands (e.g., writing a full row of memory), thus increasing thethroughput as compared to issuing each write command through the LCOMinterface. Such restore interactions with replication can continue in aloop (see loop 436) until the restore operation is complete (e.g., alldata is restored).

The throughput of the backup and restore operations (loop 424 and loop436, respectively) can be improved due to the replication of the localor LCOM commands (operation 418 and operation 430, respectively)according to the herein disclosed techniques. Embodiments of statemachines for performing such replication are discussed as pertains toFIG. 5A and FIG. 5B.

FIG. 5A is a state diagram representing a command replication statemachine 5A00 for accessing both sides of a DRAM array in hybrid memorymodules to improve data backup and restore throughput. As an option, oneor more instances of command replication state machine 5A00 or anyaspect thereof may be implemented in the context of the architecture andfunctionality of the embodiments described herein. Also, the commandreplication state machine 5A00 or any aspect thereof may be implementedin any desired environment.

The command replication state machine 5A00 represents one embodiment ofthe logic used to replicate local commands (e.g., LCOM commands)according to the herein disclosed techniques for overcoming thethroughput limitations of the non-volatile memory controller LCOMinterface used for DRAM read and write commands during data backup anddata restore operations. For example, the command replication statemachine 5A00 (e.g., included in the command replicator 176 operating atthe command buffer 126) can replicate LCOM DRAM read and/or writecommands (e.g., DRAM BC8 commands) one or more times, while optionallyaccessing alternate sides (e.g., DRAM A/B sides) and/or optionallyincrementing to a next command address. Any combination of replicationoptions is possible. In one or more embodiments, the command replicationstate machine 5A00 can operate at DRAM CK clock speeds, such that thereplicated commands can be executed at a higher throughput as comparedto the throughput of LCOM commands. Specifically, as shown, the commandreplication state machine 5A00 might be idle (see state 502) when alocal command is received and analyzed (see state 504). If the receivedlocal command is not a DRAM read or write command (e.g., a mode registersetting or MRS command) the command replication state machine 5A00 cansend the command to the DRAM without replication (see state 508). Whenthe received local command is a DRAM read or write command, the commandreplication state machine 5A00 can send the command to the target sideand/or rank address (see state 512), such side A, rank 0 (e.g., A[0]).

When the option of replicating to alternate sides is enabled (see “Yes”path of state 514), the command replication state machine 5A00 can wait(see state 552 ₁) a certain number of clock cycles (e.g., N cycles 554₁), such as DRAM CK clock cycles, before sending a replicated instanceof the command to side B, rank 0 (e.g., B[0]) (see state 516). In one ormore embodiments, the number of cycles (e.g., N cycles 554 ₁) and thecorresponding wait time can be configured based on one or more settings.When the option of replicating to alternate sides is disabled (see “No”path of state 514) and the replicated command has been sent to B[0], thecurrent address of the received command can be incremented when theincrement address option is enabled (see “Yes” path of state 518). Whenthe address is incremented and a terminal count (e.g., 128 incrementscomprising a DRAM page) has not been reached (see state 520), thecommand replication state machine 5A00 can wait (see state 552 ₂) acertain number of DRAM CK clock cycles (e.g., N cycles 554 ₂) beforesending a replicated instance of the command to the incremented addressat A[0] (see state 512). When the increment address option is disabled(see “No” path of state 518) and/or the terminal count has been reached(see “Yes” path of state 520), the command replication state machine5A00 can return to the idle state (see state 502).

The state machine 402 of the command replicator 176 can furtherreplicate to multiple ranks comprising the DRAM array as represented inthe embodiment of FIG. 5B.

FIG. 5B is a state diagram representing a command replication statemachine 5B00 for accessing multiple sides and/or multiple ranks of aDRAM array in hybrid memory modules to improve data backup and restorethroughput. As an option, one or more instances of command replicationstate machine 5B00 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the command replication state machine 5B00 orany aspect thereof may be implemented in any desired environment.

The command replication state machine 5B00 represents one embodiment ofthe logic used to replicate local commands (e.g., LCOM commands)according to the herein disclosed techniques for overcoming thethroughput limitations of the non-volatile memory controller LCOMinterface used for DRAM read and write commands during data backup anddata restore operations. For example, the command replication statemachine 5B00 (e.g., included in the command replicator 176 operating atthe command buffer 126) can replicate LCOM DRAM read and/or writecommands (e.g., DRAM BC4 commands) one or more times while optionallyaccessing alternate sides (e.g., DRAM A/B sides) and/or optionallyincrementing to a next command address. In some embodiments, the commandreplication state machine 5B00 can further replicate LCOM DRAM readand/or write commands while optionally accessing various DRAM ranksand/or optionally accommodating various burst orders and/or nibbleaddressing (e.g., for burst chop commands).

Any combination of replication options is possible. In one or moreembodiments, the command replication state machine 5B00 can operate atDRAM CK clock speeds such that the replicated commands can be executedat a higher throughput as compared to the throughput of LCOM commands.Specifically, as shown, the command replication state machine 5B00 mightbe idle (see state 502) when a local command is received and analyzed(see state 504). If the received local command is not a DRAM read orwrite command (e.g., a mode register setting or MRS command) the commandreplication state machine 5B00 can send the command to the DRAM withoutreplication (see state 508). When the received local command is a DRAMread or write command, the command replication state machine 5B00 cansend the command to the target side and/or rank address (see state 512)such side A, rank 0 (e.g., A[0]).

When the option of replicating to alternate sides is enabled (see “Yes”path of state 514), the command replication state machine 5B00 can wait(see state 552 ₁) a certain number of DRAM CK clock cycles (e.g., Ncycles 554 ₁) before sending the next replicated command. In one or moreembodiments, the number of cycles (e.g., N cycles 554 ₁) and thecorresponding wait time can be configured based on one or more settings.When two ranks are configured in the DRAM, the command replication statemachine 5B00 can send a replicated command to the second rank on thecurrent side (e.g., A[1]) (see state 532), wait (see state 552 ₃) acertain number of DRAM CK clock cycles (e.g., N cycles 554 ₃), send areplicated command to the alternate side and first rank, such as side B,rank 0 (e.g., B[0]) (see state 516), wait (see state 552 ₄) a certainnumber of DRAM CK clock cycles (e.g., N cycles 554 ₄), and send areplicated command to the alternate side and second rank, such as sideB, rank 1 (e.g., B[1]) (see state 534). When one rank is configured inthe DRAM, the command replication state machine 5B00 can send thereplicated command to B[0] following the wait corresponding to state 552₁.

When the option of replicating for alternate nibbles (e.g., lowernibble) of various burst chop (e.g., BC4) commands is enabled (see “Yes”path of state 536), the command replication state machine 5B00 canupdate to the next (e.g., lower) nibble in the burst order (see state538) and determine if both nibbles have been accessed (see state 540).When the selected nibble has not been accessed (see “No” path of state540), the command replication state machine 5B00 can wait (see state 552₂) a certain number of DRAM CK clock cycles (e.g., N cycles 554 ₂)before sending a replicated instance of the command to the selectednibble and address at A[0] (see state 512). When the option ofreplicating for alternate nibbles (e.g., lower nibble) of various burstchop commands is disabled (see “No” path of state 536) and/or allnibbles corresponding to the current address have been accessed (see“Yes” path of state 540), the current address of the received commandcan be incremented when the increment address option is enabled (see“Yes” path of state 518). When the address is incremented and a terminalcount (e.g., 128 increments comprising a DRAM page) has not been reached(see state 520), the command replication state machine 5B00 can wait(see state 552 ₂) a certain number of DRAM CK clock cycles (e.g., Ncycles 554 ₂) before sending a replicated instance of the command to theincremented address at A[0] (see state 512). When the increment addressoption is disabled (see “No” path of state 518) and/or the terminalcount has been reached (see state 520), the command replication statemachine 5A00 can return to the idle state (see state 502).

The herein disclosed techniques for improving data backup and restorethroughput in hybrid memory modules can further enable variousconnection schemes for coupling the DRAM devices and the non-volatilememory controller. For example, the higher throughput provided by theherein disclosed techniques might enable fewer chip connection paths(e.g., more parallel connections) between the DRAM devices and thenon-volatile memory controller, yet still with higher throughput ascompared to legacy architectures such as shown in FIG. 6A. Embodimentsof such reduced connection configurations implemented using the hereindisclosed techniques are discussed in FIG. 6B, FIG. 6C, and FIG. 6D.

FIG. 6A is a connection diagram of an independent connectionconfiguration 6A00 for connecting DRAM devices to a non-volatile memorycontroller for backup, restore, and/or other operations. The independentconnection configuration 6A00 can represent a portion of the connections(e.g., 108 connections) that a hybrid memory module, such as shown inFIG. 3B, might have between the DRAM devices (e.g., DRAM devices 124 ₁,DRAM devices 124 ₂) and the non-volatile memory controller 128 forbackup, restore, and/or other operations. For example, each DRAM device(e.g., DDR4-0, DDR4-1, . . . , to DDR4-17) might have a dedicated buscomprising six signals (e.g., DQS, DQ[3:0], CS) between the DRAM deviceand the non-volatile memory controller 128. Specifically, the DQSx andDQx[3:0] signals are shown in FIG. 6A for illustrative purposes.

Using the herein disclosed techniques for improving data backup andrestore throughput in hybrid memory modules, alternative connectionschemes for coupling the DRAM devices and the non-volatile memorycontroller can be implemented to simplify chip routing, reduceconnection trace area and associated chip costs, and other benefits.Such alternative connection schemes (e.g., see FIG. 6B, FIG. 6C, andFIG. 6D) are possible due to the throughput improvements of the hereindisclosed techniques.

FIG. 6B is a connection diagram of a first dual connection configuration6B00 having parallel A-side and B-side connections as implemented inhybrid memory modules with improved data backup and restore throughput.As an option, one or more instances of first dual connectionconfiguration 6B00 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the first dual connection configuration 6B00 orany aspect thereof may be implemented in any desired environment.

As shown in the embodiment comprising the first dual connectionconfiguration 6B00, two DRAM devices can have a shared path fortransmitting data signals (e.g., DQ0[3:0], DQ1[3:0], etc.) and datastrobe signals (e.g., DQS0, DQS1, etc.) to and/or from the non-volatilememory controller 128. As shown, a given shared path can be coupled to aDRAM device on the A-side of the module and a DRAM device on the B-sideof the module. In such cases, the herein disclosed techniques can usethe chip select (CS) signal of each DRAM device (not shown) and theoption of replicating commands for alternate sides to read to and/orwrite from the DRAM devices on each of the shared paths. Some DRAMdevices (e.g., DDR4-4, DDR4-13) might not use a shared path based on theDRAM configuration.

FIG. 6C is a connection diagram of a second dual connectionconfiguration 6C00 having dual parallel DRAM connections as implementedin hybrid memory modules with improved data backup and restorethroughput. As an option, one or more instances of second dualconnection configuration 6C00 or any aspect thereof may be implementedin the context of the architecture and functionality of the embodimentsdescribed herein. Also, the second dual connection configuration 6C00 orany aspect thereof may be implemented in any desired environment.

As shown in the embodiment comprising the second dual connectionconfiguration 6C00, two DRAM devices can have a shared path fortransmitting data signals (e.g., DQ0[3:0], DQ5[3:0], DQ1[3:0], etc.) anddata strobe signals (e.g., DQSA0, DQSA5, etc.) to and/or from thenon-volatile memory controller 128. As shown, a given shared path can becoupled to a DRAM device on the A-side of the module and a DRAM deviceon the B-side of the module. In such cases, the herein disclosedtechniques can use the chip select (CS) signal of each DRAM device (notshown) and the option of replicating commands for alternate sides toread to and/or write from the DRAM devices on each of the shared paths.Some DRAM devices (e.g., DDR4-4, DDR4-13) might not use a shared pathbased on the DRAM configuration.

FIG. 6D is a connection diagram of a quad connection configuration 6D00having quad parallel DRAM connections as implemented in hybrid memorymodules with improved data backup and restore throughput. As an option,one or more instances of quad connection configuration 6D00 or anyaspect thereof may be implemented in the context of the architecture andfunctionality of the embodiments described herein. Also, the quadconnection configuration 6D00 or any aspect thereof may be implementedin any desired environment.

As shown in the embodiment comprising the quad connection configuration6D00, four DRAM devices can have a shared path for transmitting datasignals (e.g., DQ0[3:0], DQ1[3:0], etc.) and data strobe signals (e.g.,DQS0, DQS2, etc.) to and/or from the non-volatile memory controller 128.As shown, a given shared path can be coupled to a DRAM device from eachrank (e.g., rank[0], rank[1]) on the A-side of the module, and a DRAMdevice from each rank (e.g., rank[0], rank[1]) on the B-side of themodule. In such cases, the herein disclosed techniques can use the chipselect (CS) signal of each DRAM device (not shown) and the option ofreplicating commands for alternate sides and/or multiple ranks to readto and/or write from the DRAM devices on each of the shared paths. Anynumber of DRAM devices (e.g., see DDR4-4 and DDR4-13) can use a sharedpath, and/or not use a shared path, based on the DRAM configuration.

FIG. 7A is a diagram of a proprietary access subsystem 7A00 asimplemented in systems for enhancing non-volatile memory controllerresource access. As an option, one or more instances of proprietaryaccess subsystem 7A00 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the proprietary access subsystem 7A00 or anyaspect thereof may be implemented in any desired environment.

In some implementations of the command buffer 126, such as those definedby JEDEC, a set of control settings 722 stored in the control settingregisters 182 can be accessed using the control setting access logic181. The control setting registers 182 might have a standard registerspace 716 (e.g., JEDEC-defined function spaces 0-7) and a vendorregister space 718 (e.g., JEDEC-defined function spaces 8-15). In somecases, the control setting access logic 181 can provide a direct access712 to the control setting registers 182. In other cases, the controlsetting access logic 181 can provide an indirect access 714 to thecontrol setting registers 182. For example, function space 0 might beaccessed directly, yet reads and/or writes to function spaces 1-15 mightbe accessed through function 0 (e.g., by a combination of F0RC4x,F0RC5x, and F0RC6x writes). Further, the host memory controller 105 andthe non-volatile memory controller 128 can have full access, to at leastthe standard register space 716, when in a host control mode and anon-volatile memory controller control mode (e.g., NVC control mode),respectively. Yet, in the host control mode, the non-volatile memorycontroller 128 might have restricted access to a protected registerspace 185. For example, to avoid conflicts among the host commands 171issued by the host memory controller 105 and the local commands 172issued by the non-volatile memory controller 128 in host control mode,the control setting access logic 181 might allow the non-volatile memorycontroller 128 access to a subset of the standard register space 716(e.g., control word locations F0RC07, F4RC00, and F4RC02), yet no accessto the protected register space 185. Such restricted access can resultin long latencies and high power consumption when a backup and/orrestore event is invoked since the non-volatile memory controller 128 islimited in its ability to prepare certain settings (e.g., in theprotected register space 185) in advance of such events.

The proprietary access subsystem 7A00 shown in FIG. 7A addresses suchresource access restrictions for the non-volatile memory controller 128,yet does not impact the host memory controller 105 resource access.Specifically, in one or more embodiments, the proprietary accesssubsystem 7A00 can comprise a proprietary access engine 184 to interpretone or more local commands 172 from the non-volatile memory controller128 to access the protected register space 185 while still in the hostcontrol mode. Specifically, in some embodiments, the proprietary accessengine 184 might receive a sequence of the local commands 172 thattrigger a proprietary mode such that subsequent instances of localcommands 172 can be interpreted as instances of proprietary accesscommands 188 for accessing the protected register space 185.

In one or more embodiments, the proprietary access engine 184 cancomprise a set of proprietary control setting access logic 704, based inpart on the control setting access logic 181, to interpret theproprietary access commands 188 to write to and/or read from theprotected register space 185. Further, the proprietary access engine 184might comprise a command router 702 to route the local commands 172 tothe control setting access logic 181 and route the proprietary accesscommands 188 to the proprietary control setting access logic 704.Further, the command router 702 might comprise a set of proprietary modetrigger logic 726 to decode received instances of local commands 172 todetermine when the proprietary mode can be enabled. Also, in someembodiments, the proprietary access engine 184 can comprise an accessarbiter 706 to allow only one of the host commands 171 and proprietaryaccess commands 188 access to the control setting registers 182 at agiven time.

The proprietary access subsystem 7A00 presents merely one partitioning.The specific example shown is purely exemplary, and other partitioningis reasonable. A technique for expanding the non-volatile memorycontroller resource access, yet not impact host memory controllerresource access, implemented in such systems, subsystems, andpartitionings is shown in FIG. 7B.

FIG. 7B illustrates a proprietary access protocol 7B00 as used in hybridmemory modules for enhancing non-volatile memory controller resourceaccess. As an option, one or more instances of proprietary accessprotocol 7B00 or any aspect thereof may be implemented in the context ofthe architecture and functionality of the embodiments described herein.Also, the proprietary access protocol 7B00 or any aspect thereof may beimplemented in any desired environment.

The proprietary access protocol 7B00 presents one embodiment of certainsteps for expanding the non-volatile memory controller resource access,yet not impact host memory controller resource access, during a hostcontrol mode. In one or more embodiments, the steps and underlyingoperations shown in the proprietary access protocol 7B00 can be executedby the command buffer 126 disclosed herein. As shown, the proprietaryaccess protocol 7B00 can commence with receiving local commands (seestep 736), such as LCOM commands defined by JEDEC. The received localcommands can be used to determine if a proprietary mode can be enabled(see decision 738). In one or more embodiments, the proprietary mode canallow the local commands to be interpreted, routed, and processed asproprietary access commands (e.g., proprietary access commands 188). Insuch cases, when the proprietary mode is enabled and the received localcommand is a proprietary access command (see decision 740), theproprietary access command can be routed (e.g., by command router 702)for proprietary access command processing (see step 744). When theproprietary mode is not enabled and/or the received command is not aproprietary access command, the received local command can be routed tothe architected control setting resources (see step 742), such as thecontrol setting access logic 181 and/or the control setting registers182. In such cases, the received local command may not be able to accessthe protected register space 185 in the control setting registers 182when in host control mode. Yet, when proprietary mode is enabled and aproprietary access command is received, the proprietary access protocol7B00 might further determine (e.g., by access arbiter 706) whether ahost command is being executed (see decision 746). When a host commandis being executed, a certain wait delay can transpire (see step 748)before returning to check the host command execution status. When thereare no host commands executing, the proprietary access command can beexecuted (see step 750), for example, by the proprietary control settingaccess logic 704, to access the protected register space 185 during hostcontrol mode, according to the herein disclosed techniques.

FIG. 8A depicts a mode register controller subsystem 8A00 used in hybridmemory modules for enhancing the programmability of the DRAM moderegister settings in a non-volatile memory controller control mode. Asan option, one or more instances of mode register controller subsystem8A00 or any aspect thereof may be implemented in the context of thearchitecture and functionality of the embodiments described herein.Also, the mode register controller subsystem 8A00 or any aspect thereofmay be implemented in any desired environment.

As shown in FIG. 2B, different mode register settings (see message 261,message 275, and message 285) in the DRAM devices might be required in ahost control mode and/or an NVC control mode due to various reasons,such as different operating frequencies and capabilities in therespective modes. The time and power required to execute the moderegister setting or MRS commands can impact the performance of thehybrid memory module when switching between a host control mode and anNVC control mode, such as when invoked during data backup and datarestore operations. In some cases, the non-volatile memory controller128 might read some of the mode register setting data from the DRAMdevices to reprogram the mode register settings for the DRAM devices(e.g., when entering and/or leaving an NVC control mode), yet some moderegister settings data (e.g., certain register bits) are not accessibleby the non-volatile memory controller 128. Such restricted accessibilityto protected register bits can limit the ability of the non-volatilememory controller 128 to modify the mode register settings (e.g.,comprising accessible register bits and protected register bits) of theDRAM devices when in NVC control mode, yet not overwrite certainregister settings established when in host control mode. Further,writing the DRAM mode register settings (e.g., using MRS commands) usingthe LCOM interface of the non-volatile memory controller 128 can becostly in terms of execution latencies and power consumption, as theLCOM protocol can require 128 DRAM clock cycles per DRAM command (e.g.,read, write, etc.) issued from the non-volatile memory controller 128.

The mode register controller subsystem 8A00 shown in FIG. 8A addressessuch limitations by enhancing the programmability of the DRAM moderegister settings in a non-volatile memory controller control mode(e.g., NVC control mode), such as when invoked during data backup anddata restore operations. Specifically, in one or more embodiments, themode register controller subsystem 8A00 can receive host commands 171from the host memory controller 105, receive local commands 172 from thenon-volatile memory controller 128, and issue DRAM commands (e.g., DRAMcommands 174 ₁, DRAM commands 174 ₂) to the DRAM devices (e.g., DRAMdevices 124 ₁, DRAM devices 124 ₂). For example, such DRAM commands canbe mode register setting (MRS) commands issued by the host memorycontroller 105 (e.g., during host control mode) and/or by thenon-volatile memory controller 128 (e.g., during NVC control mode).Further, the mode register controller subsystem 8A00 can comprise a moderegister controller 192 to capture (e.g., “snoop”) a set of capturedmode register settings 194 from the host commands 171 and generatecertain generated mode register setting commands (e.g., generated moderegister setting commands 196 ₁, generated mode register settingcommands 196 ₂) based on the captured mode register settings 194.

More specifically, in some embodiments, the captured mode registersettings 194 can be extracted from the host commands 171 by a MRScommand decoder 804. Further, the generated mode register settingcommands can be generated at least in part from the captured moderegister settings 194 by an MRS command generator 802. In one or moreembodiments, the generated mode register setting commands can be issuedto the DRAM devices from the command buffer 126, responsive to receivingcertain instances of the local commands 172. In some embodiments, thecaptured mode register settings 194 can be modified before being used toproduce the generated mode register setting commands. For example,certain register bits might need to be toggled for NVC control mode, yetother register bits established while in host control mode (e.g.,related to timing training) might need to remain in their current state.As another example, certain events occurring during the NVC control modemight require certain mode register settings to be different afterleaving the NVC control mode as compared to when entering the NVC mode(e.g., based on temperature, termination mode, termination values,etc.). Further, in some cases, the captured mode register settings 194can be captured when the host memory controller 105 initializes thehybrid memory module comprising the mode register controller subsystem8A00. The captured mode register settings 194 can further be accessedand/or modified by the host memory controller 105 or the non-volatilememory controller 128 (e.g., for reading, writing, modifying, etc.).Also, the captured mode register settings 194 might be stored in thecontrol setting registers 182 of the command buffer 126.

The mode register controller subsystem 8A00 presents merely onepartitioning. The specific example shown is purely exemplary, and otherpartitioning is reasonable. A technique for improving latencies andpower consumption when switching between a host control mode and an NVCcontrol mode implemented in such systems, subsystems, and partitioningsis shown in FIG. 8B.

FIG. 8B is a diagram showing interactions 8B00 in hybrid memory modulesthat implement a mode register controller for enhancing theprogrammability of the DRAM mode register settings in a non-volatilememory controller control mode. As an option, one or more instances ofinteractions 8B00 or any aspect thereof may be implemented in thecontext of the architecture and functionality of the embodimentsdescribed herein. Also, the interactions 8B00 or any aspect thereof maybe implemented in any desired environment.

As shown in FIG. 8B, the interactions 8B00 pertain to the earlierdescribed components comprising the host memory controller 105, thecommand buffer 126, the non-volatile memory controller 128, thecollective set of DRAM devices 124, and the flash memory devices 134,according to the herein disclosed techniques for improving latencies andpower consumption when switching between a host control mode and an NVCcontrol mode, such as invoked by data backup and data restore events. Inone or more embodiments, the interactions 8B00 can be implemented usingthe mode register controller subsystem 8A00. As shown, the commandbuffer 126 might receive one or more host commands from the host memorycontroller 105 (see message 812) in a host control mode to be forwardedto the DRAM devices 124 (see message 814). In some cases, the commandbuffer 126 can decode any MRS commands included in the host commands(see operation 816) and capture the mode register settings from thedecoded MRS commands (see operation 818). For example the captured moderegister settings might represent the settings that have been determined(e.g., using timing training, etc.) to be appropriate for the hostcontrol mode.

After a time lapse 822, a data backup or restore event signal might bereceived by the non-volatile memory controller 128 (see operation 824).In response, control can be provisioned to the non-volatile memorycontroller 128 by, for example, writing to certain control registersettings of the command buffer 126 (see message 826). In some cases, thenon-volatile memory controller 128 might need to modify certain moderegister settings to prepare the DRAM devices 124 for NVC control.Specifically, certain register bits might need to be toggled for NVCcontrol mode, yet other register bits established while in host controlmode (e.g., related to timing training) might need to remain in theircurrent state. According to the herein disclosed techniques, suchmodifications can be executed using the earlier captured mode registersettings (see grouping 836). More specifically, the non-volatile memorycontroller 128 can issue certain mode register settings commands (seemessage 828) to the command buffer 126 indicating the register bits tobe set for operations during NVC control mode. The command buffer 126can apply the received register settings to the captured mode registersettings to generate a set of modified captured mode register settings(see operation 830). The command buffer 126 can use the modifiedcaptured mode register settings to generate a set of MRS commands (seeoperation 832 ₁) that can be issued directly to the DRAM devices 124(see message 834 ₁).

When the non-volatile memory controller 128 completes the execution ofthe backup or restore operations (see operation 838), a process forrestoring the host mode register settings can commence (see grouping840). Specifically, the captured mode register settings might bemodified (see operation 842) based on certain information (e.g., changesin chip temperature, DRAM configuration, etc.). The non-volatile memorycontroller 128 can then issue a trigger command to the command buffer126 (see message 844) to invoke the generation of MRS commands (seeoperation 832 ₂) based on the captured and/or modified mode registersettings, according to the herein disclosed techniques. In some cases,the trigger command may not be required to invoke the MRS commandgeneration. The generated MRS command can then be issued by the commandbuffer 126 to the DRAM devices 124 (see message 834 ₂) prior toprovisioning control back to the host memory controller 105.

The techniques illustrated in FIG. 8B and described herein enable themodification of the mode register settings of the DRAM devices 124 whenin NVC control mode, yet not overwrite certain register settingsestablished when in host control mode. Further, by reducing the numberof commands from the non-volatile memory controller in restoring thehost mode register settings, the herein disclosed techniques improve thelatencies and power consumption when switching from the NVC control modeto the host control mode. For example, in some embodiments, the commandbuffer 126 can issue a set of generated MRS commands sixteen timesfaster as compared to issuing the MRS commands using the LCOM interfaceof the non-volatile memory controller 128. Further, as another example,all seven MRS commands (e.g., for MRS0 to MRS6) can be completed in 49DRAM CK clocks per rank, including a tMRD wait.

Additional Examples

It should be noted that there are alternative ways of implementing theembodiments disclosed herein. Accordingly, the embodiments and examplespresented herein are to be considered as illustrative and notrestrictive, and the claims are not to be limited to the details givenherein, but may be modified within the scope and equivalents thereof.

ADDITIONAL EMBODIMENTS OF THE DISCLOSURE

In the foregoing specification, the disclosure has been described withreference to specific embodiments thereof. It will, however, be evidentthat various modifications and changes may be made thereto withoutdeparting from the broader spirit and scope of the disclosure. Forexample, the above-described process flows are described with referenceto a particular ordering of process actions. However, the ordering ofmany of the described process actions may be changed without affectingthe scope or operation of the disclosure. The specification and drawingsare, accordingly, to be regarded in an illustrative sense rather than ina restrictive sense.

1. (canceled)
 2. A memory module comprising: a non-volatile memorydevice; a dual-port volatile memory device; and a memory controllerdevice to control the non-volatile memory device and coupled to a firstport of the dual-port volatile memory device via a direct datatransmission path, wherein data transferred over the direct datatransmission path bypasses a data buffer coupled to a second port of thedual-port volatile memory device.
 3. The memory module of claim 2,wherein the data buffer is powered off during a backup and restoreoperation on the memory module and backup and restore data istransferred between the dual-port volatile memory device and the memorycontroller over the direct data transmission path.
 4. The memory moduleof claim 2, wherein the dual-port volatile memory device is configuredto enter a mode where a delay locked loop is turned off during a backupand restore operation when the direct data transmission path is in use.5. The memory module of claim 2, wherein the first port is not used andthe second port is used when transmitting data between the dual-portvolatile memory device and the non-volatile memory device.
 6. The memorymodule of claim 2, wherein the memory controller device to use thedirect data transmission paths to transmit the data between thedual-port volatile memory device and the non-volatile memory deviceduring a memory controller device control mode.
 7. The memory module ofclaim 6, wherein the memory controller device control mode is invoked inresponse to at least one of a data backup event or a data restore event.8. The memory module of claim 2, wherein the direct data transmissionpath to transmit one or more electronic signals between the dual-portvolatile memory device and the non-volatile memory device.
 9. The memorymodule of claim 8, wherein the one or more electronic signals compriseat least one of a data signal, a chip select signal, or a data strobesignal.
 10. A memory module comprising: a flash memory device; adual-port dynamic random access memory (DRAM) device; and a memorycontroller device to control the flash memory device and coupled to afirst port of the dual-port DRAM device via a direct data transmissionpath, wherein data transferred over the direct data transmission pathbypasses a data buffer coupled to a second port of the dual-port DRAMdevice.
 11. The memory module of claim 10, wherein the data buffer ispowered off during a backup and restore operation on the memory moduleand backup and restore data is transferred between the dual-port DRAMdevice and the memory controller over the direct data transmission path.12. The memory module of claim 10, wherein the dual-port DRAM device isconfigured to enter a mode where a delay locked loop is turned offduring a backup and restore operation when the direct data transmissionpath is in use.
 13. The memory module of claim 10, wherein the firstport is not used and the second port is used when transmitting databetween the dual-port DRAM device and the flash memory device.
 14. Thememory module of claim 10, wherein the memory controller device to usethe direct data transmission paths to transmit the data between thedual-port DRAM device and the flash memory device during a memorycontroller device control mode.
 15. The memory module of claim 14,wherein the memory controller device control mode is invoked in responseto at least one of a data backup event or a data restore event.
 16. Thememory module of claim 10, wherein the direct data transmission path totransmit one or more electronic signals between the dual-port DRAMdevice and the flash memory device.
 17. The memory module of claim 16,wherein the one or more electronic signals comprise at least one of adata signal, a chip select signal, or a data strobe signal.
 18. A memorymodule comprising: a non-volatile memory device; a dual-port volatilememory device; a data bus comprising a data buffer; and a memorycontroller device to control the non-volatile memory device and coupledto a first port of the dual-port volatile memory device via a directdata transmission path, wherein data transferred from the dual-portvolatile memory device to the non-volatile memory device over the directdata transmission path or from the non-volatile memory device to thedual-port volatile memory device over the direct data transmission pathbypasses the data buffer of the data bus coupled to a second port of thedual-port volatile memory device.
 19. The memory module of claim 18,wherein the data buffer of the data bus is powered off during a backupand restore operation on the memory module and backup and restore datais transferred between the dual-port volatile memory device and thememory controller over the direct data transmission path.
 20. The memorymodule of claim 18, wherein the dual-port volatile memory device isconfigured to enter a mode where a delay locked loop is turned offduring a backup and restore operation when the direct data transmissionpath is in use.
 21. The memory module of claim 18, wherein the firstport is not used and the second port is used when transmitting databetween the dual-port volatile memory device and the non-volatile memorydevice.